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SVSTFM AND Mi^THOn FOR EFF iriFNTl V FNCOPING AN IMAGE BY 
ppinPTTTZlNG GRQ TTPS OF SPATI>1 T V rORRELATED 
rnFFFiriFNTS BAS l?n ON AN ACTIVITV MEASURE 

REFERENCE TO RELATED APPLICATION 

This ^pUcation is a divisional appUcation under 37 C.F.R. 1.53(b) of co-pending 
United States Patent Application Number 09/628,106, filed July 27, 2000, the contents of 
which are expressly incorporated herein by reference. 

FIELD OF THE INVENTION 

This invention pertains to encoding and decoding data to reduce its byte size. 
More particularly the invention pertains to an efficient system for post processing 
transformed data coefficients (e.g., wavelet transformed image data). 

BACKGROUND OF THE INVENTION 

In the information age, the exchange of a variety of forms of multimedia data is 
commonplace and ever increasing. The exchanged data may, for example, comprise 
images, audio, or time series data representing scientific or business related informalioa 

Various schemes of encoding data are known. A key purpose of encoding data is 
to 'compress' the data, i.e., to reduce the byte size of the data. This is desirable for 
reasons including the reduction of memory space required to store the data, and reduction 
of the time required to transmit the data through a communication channel having a certain 
finite bandwidth. The byte size can be expressed as bits per sample, or as is conventional in 
the case of image data, as bits per pixel. 

Two classes of encoding methods are lossless and lossy. The former, more 
conservative approach endeavors to preserve every detail of the input data in the encoded 
form. IdeaUy the decoded version wouW be an indistinguishable repKca of the input data. 
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In the case of lossy data encoding (compression), the level to which the detail of 
the image is preserved can be selected and there wiU be a tradeoff between the level of 
detail preserved and the byte size of the resulting encoded data. Often in using lossy data 
encoding, one strives to obtain a level of detail preservation such that the differences 
5 between a decoded version and the original are inq,erceptible. Judgments about the design 
and configuration of the lossy encoder to achieve imperceptible differences will be made in 
consideration of human perception models (e.g. hearing, or visual). A good lossy encoder 
and corresponding decoder will yield a decoded data set which may be distinguished firom 
the original data set by rigorous scientific analysis but is indistinguishable to a human 
10 observer when presented in an intended format (e.g., screen resolution in the case of 
images). 

One class of data encoding methods (appUcable to image data) is transform coding. 
Generally speaking, in transform coding an ordered data set is projected onto an 
orthogonal set of basis fanctions to obtain a set of transformed data coefficients (inner 

15 products). The traditional type of transform coding derives firom Fourier analysis. In 
Fourier based techniques, a data set is projected onto a set of fiinctions derived fi:om 
sinusoidal fimctions. The old JPEG standard (ISO/IEC 10928-1) is an example of a 
transform encoding method based on Fourier analysis. The old JPEG standard specifies a 
set of transform matrices which are discrete representations of products of a cosine 

20 fimction with a horizontal coordinate dependant argument and a cosine fimction with a 
vertical coordinate dependant argument. These basis fimctions are appUed to analyze 8 by 

8 pixel blocks of an input image. 

A shortcoming of the Fourier based techniques, which prompted workers in the 
data coinpression art to take up other techniques as will be described shortly, is the feet 
25 that the sinusoidal fimction repeat indefinitely out to plus and minus infinity, whereas data 
sets which are encoded are localized in the time (or spatial) domain and have features 
which are fiirther localized within the data set. Given the unbounded domain ofFourier 
bases fimctions and the periodic nature of data sets to be encoded over long intervals (or 
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spans) one is led to segment the signal (e.g., into the aforementioned 8 by 8 blocks) in 
order to obtain a more efficient encoding. Unfortunately, this leads to abrupt jumps in the 
decoded version of the signal at edges between the segments. In the image processing art 

this is known as blocking effects. 
5 Newer classes of transform methods employ basis fonctions which are inherently 

focalized in the spatial domain. MathematicaUy speaking they are compactly supported. 

One example of the newer type of transform method is the wavelet based techniques. 

Wavelet based techniques employ a set of basis fimctions comprising a mother wavelet and 

a set of chad wavelets derived from the mother wavelet by applying different time (or 
10 spatial) domain shifts and dilations to the mother wavelet. A wavelet basis set comprising 

a set of functions with localized features at different characteristic scales, is better suited to 

encode data sets such as image or audio data sets which have fine, course and intermediate 

features at different locations (times). 

Multilevel wavelet decomposition is an iterative process, namely multi-resolutional 
15 decomposition. At each iteration a lower frequency set of transformed data coefficients 

generated by a prior iteration is again refined to produce a substitute set of transformed 
data coefficients including a lower spatial frequency group and a higher spatial frequency 

gjmup, called subbands. 

Returning to the matter of lossy encoding, whether it be Fourier, wavelet or 

20 otherwise based, the manner in which the reduction in the byte size (with the associated 
loss of detail) is achieved, according to the common prior art ^proach, is by quantizing 
the transformed data coefficients. Depending on the quantization and coding methods 
used, this step can be a computationally intensive operation which may account for up to 
70% of the conputational expense of a wavelet based encoder. Quantizing involves 

25 adjustmg downward the resolution with which the value of the transformed data 

coefficients are recorded, so that they can be recorded using fewer bits. In the case of 
image data, transformed data coefficients associated with basis fiinction that depend on 
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finer details in the data wiU be quantized with less resolution. TTxe decision to do this is 
based on current understanding of human visual perception. 

In connection with perception as it relates to lossy iinage encoding it bears 
mentioning that there are quantitative n^s such as SNRwWch aim at characteri^ 

the fideUty of an encoded and decoded image to the original, however these measures are 
not ahvays in accord with human assessments of the quaUty. 

All of the quantizing values can be adjusted to yield lower image fidelity and 
correspondingly reduced byte size. Quantizing values that are used for an entire image can 
be varied, or diferent sets of quantizing values that are used for separate blocks of an 

image can be adjusted. 

The new JPEG 2000 standard suggests providing "progressive transmission" 
whereby an encoded image can be transmitted at various resolutions. This is intended to 
afford adaptation to the display of the cUent receiving the image. According to the 
standard, this will be accomplished by a certain arrangement of the bit stream. 

The predominant appUcation for transform encoding is for storing data on, and 
sending data over the Internet. Extensk>n of the Internet to wireless physical networks is 
currently in the early stages. Wireless networks generaUy have less bandwidth than fiber 
optic or copper Internet networks, therefore slower web browsing and long downloads can 
be expected for the same byte size transform encoded data (e.g., images). 

Portable wireless devices generally have a low processing power and are therefore 
ill suited to perform conq)utationally intensive operations such as quantizing and complex 
entropy coding. 

Wireless networks are subject to connection instabiUty. A comiection may M 
vMe an image is being transmitted and decoded. Prior art decoding schemes paint images 
in raster scan pattern, either in one pass or multiple passes, yielding successively better 
resolution. In the former case, if the comiection is broken, part of the area of the image wiU 
be missing. In the latter case, the image would be left with an unsightly boundary between 
an upper high-quality area and a lower poor-quaUty area. 
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What is needed is a system and method for encoding and decoding data which witt 
achieve the highest possible level of fidelity of the decoded data with respect to the original 
data for a given allocation of bits per sample in the encoded representation with minimal 
conq)utational costs. 

5 

BRIEF DESCRIPTION OF THE FIGURES 

The features of the invention beUeved to be novel are set forth in the claims. The 
invention itself however, may be best understood by reference to the foUowing detailed 
description of certain exemplary embodiments of the invention, taken in conjunction with 
1 0 the accompanying drawings in which: 

FIG. 1 is a schematic of a communication network and system according to a 

preferred embodiment of the present invention. 

no. 2 is a block diagram of a portable wireless communication device shown in 
HG. 1 according to a preferred embodiment of the present invention. 
15 FIG. 3 is a block diagram of an image encoder according to a preferred 

embodiment of the present invention. 

FIG. 4 is a block diagram of an image decoder according to a preferred 

embodiment of the present invention. 

FIG. 5 is an exemplary data structure representation of the output from a three 

20 level discrete wavelet transformatioa 

HG. 6 is a flow diagram of a process performed by the spatial activity analyzer 
shown in FIG. 3 according to a preferred embodiment of the invention. 

FIG. 7 is a flow diagram of a process performed by the spatial prioritizer/selector 
shown in FIG. 3 according to a preferred embodiment of the inventioa 
25 HG. 8 is a flow diagram of a process performed by the spatial coefficient grouper 

shown in FIG. 3 according to a preferred embodiment of the invention. 

FIG. 9 is a flow diagram of a process performed by the bit stream formatter shown 
in FIG. 3 according to a preferred embodiment of the invention. 
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FIG. 10 is a flow diagram of a process performed by the bit stream parser shown in 
no. 4 according to a preferred embodiment ofthe invention. 

FIG. 1 1 is a flow diagram of a process performed by the spatial coefficient 
reorganizer shown in FIG. 4 according to a preferred embodiment ofthe invention 
5 FIG. 12 is a first portion of a flow diagram of a process performed by the spatial 

activity analyzer shown in FIG. 3, according to an alternative preferred embodiment ofthe 
present invention. 

FIG. 13 is a second portion of a flow diagram of a process performed by the spatial 
activity analyzer shown in HG. 3, according to an alternative preferred embodiment ofthe 

10 present invention. 

FIG. 14 is a flow diagram of a process performed by the spatial prioritizer/selector 
shown in FIG. 3, according to an alternative preferred embodiment ofthe invention. 

FIG. 15 is a flow diagram of a process performed by the spatial coefficient grouper 
shown in FIG. 3, according to an alternative preferred embodiment ofthe invention. 
15 FIG. 16 is a flow diagram of a process performed by the bit stream formatter 

shown in FIG. 3 according to an alternative preferred embodiment ofthe invention. 

FIG. 17 is a flow diagram of a process performed by the bit stream parser shown in 
FIG. 4 according to an alternative preferred embodiment ofthe invention. 

FIG. 1 8 is a flow diagram of a process performed by the spatial coefficient 
20 reorganizer shown in FIG. 4 according to an alternative preferred embodiment ofthe 
invention. 

HG. 19 is a flow diagram of a process performed by the spatial activity analyzer 
shown in FIG. 3 according to an alternative preferred embodiment ofthe present inventioa 
FIG. 20 is a flow diagram of a process performed by the image encoder shown in 
25 FIG. 3 or the image decoder shown in FIG. 4 according to an alternative preferred 
embodiment of the invention. 
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FIG. 21 is a flow diagram of a process that can be performed by the bit stream 
formatter shown in HG. 3 according to an alternative preferred embodiment ofthe 
invention. 

5 DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 

While this invention is susceptible of embodiment in many different forms, there are 
shown in the drawings and will herein be described in detaU specific embodiments, with the 
understanding that the present disclosure is to be considered as an example ofthe 
principles ofthe invention and not intended to limit the invention to the specific 
10 embodiments shown and described. Further, the terms and words used herein are not to be 
considered limiting, but rather merely descriptive. In the description below, like reference 
numbers are used to describe the same, similar, or corresponding parts in the several views 
ofthe drawii^s. 

According to preferred embodiments ofthe present invention, a system and method 
15 operate to encode and decode data to achieve the highest possible level of fidelity ofthe 
decoded data with respect to the original data for a given altocation of bfts per sample in 
the encoded representation. 

Further, in the case of image data, a system and method are provided for encoding 
and decoding images to achieve the highest possible level of fideUty ofthe decoded image 
20 with respect to the original data for a given allocation of bits per sanq)le in the encoded 
representation. 

Further, in the case of image data, a system and method are provided for encoding 
and decoding images to achieve the highest possible level of fidelity of the decoded image 
with respect to the original data for a given aUocation of bits per sample in the encoded 
25 representation without the need to use a computationally intensive quantizing and coding 
operations to reduce the byte size ofthe encoded image. 

Further a system and method are provided whereby a data set may be encoded 
once to obtain an encoded representation, and various byte size parts ofthe encoded 

-7- 



Atty. Docket: CM03347JD01 
Express Maa No. EV 116540 955 US 



^epreseoWion nuy be selectively decoded to obtain versions with different leveb of fide!«y 
vrt^reby for ««h byte size part decoded a high level of fideWy to the original da^ 
achi^ for the given the byte size of the decoded part relative to the nun*er otsan,ples 

inthe data set 

5 Further, a system and method are provided whereby a data set may be encoded into 

asequenced multipart data structureonce, and partsofthe multipart data stn^cture^ 

be sequentially decoded to iteratively refine a decoded version of the data set such that 
after each part of the data structure is decoded and used to refine (update) the decoded 
version the decoded version will have a high subjective fidelity and/or SNR for the byte 
10 size of the parts of the encoded data structure that were decoded. 

Further a method and system are provided for encoding images that provides for 
trading off image quality for encoded byte size that does not involve the computationally 
intensive process of dequantizing nor the intensive process of decoding. 

Further a method and system are provided for encoding and decoding images 
which will lead to more presentable partially decoded images in the case that a comiection 
through which image data is being transferred is broken. 

v^^rr^Ur y Network A ^.^rHmp to an Embodiment of the Invention 

Referring to FIG. 1 a schematic of the computer network 100 used in connection 
withanembodimentoftheinventionisshown. An image source server 102 is comiected 
to the Internet 104 through a first bidirectional data link 106. A wireless network 
gateway server 108 is connected to the Internet through a second bi-directional data link 
1 10. The wireless network gateway server 108 is connected to a wireless network 1 12 
throughathird bi-directional data link 114. The three bi-directk,nal data links 106. 110, 
and 1 14 can comprise, for example, wired and wireless communication links such as 
dedicated lines, PubUc Switched Telephone Network (PSTN) lines, Tl lines, radio 
commumcationlinks,microwavelinks,andsatellitelinks. The wireless network comprises 
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a transceiver 1 1 6 which establishes a wireless communication channel with a portable 
wireless communication device 118. The portable wireless communication device 118, in 
one exemplary application, can comprise a portable wireless cUent device 118. 

The image source server 102 may for exan^k con^rise a World Wide Web server 
which serves HTML documents comprising images. Tb, wireless network gateway server 
can be a wireless ^pUcation protocol (WAP) server. 

The fanage source server 102 and the wireless network gateway server 108 may 
take the form of a standard PC computers, comprising BIOS ROM, RAM, hard drives, 
keyboard, mouse, display screen, and a removable media drive. THe removable media dnve 
rmy be used to load data encoding and/or decoding software from a computer readable 
medium 120 for configuring one or both of the servers 102, 108 to perform processes of 
the present invention described below. The removable media may be for example an 
optical disk (e.g.. CDROM). a magnetic media disk (e.g. 3.5" 1 .44 Meg Disk) 
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p^^^ior y Pnrtahle Wire k «« rnmmunication device Accord in p to a Preferred 
TTmlviHinnent of the Tnvention 

FIG. 2 shows an internal schematic of the portable wireless communication device 
118(FIG 1) according to an embodiment of the invention. A microprocessor 202 
executesprogramsfordea,ding images. Random access memory (RAM) 204 provides a 
work space for use by the microprocessor 202. Read only memory (ROM) 206 stores 
programs for decoding and/or encoding images. A display 208, which is used to display 
decodedimages,isoperativelycoupledtoadisplaydriver210. Auserinterfece212is 
used for accepting user input, such as a request to view a particular image stored on the 
image source server 102 (FIG. 1), for further processing a particular region of a displayed 
irriage or a command to transmit an image to another communication device (not shown). 
An image processing application specific integrated circuit (IP ASIC) 220, and an image 
processing digital signal processor (IP DSP) 222 are also provided. An image acquisition 
device 224, which can take the form of a digital camera, is also integrated into the wireless 

communication device 118. 

A communication interfece 214, a transceiver 216, and an antenna 218 are also 

provided. 

A bi-directional system bus 226 couples the microprocessor 202, RAM 204, ROM 
206, display interfece 210, user interfece 212, communication interfece 214, IP ASIC 220, 
IP DSP 222, and image acquisition device 224. 

The IP ASIC 220, IP DSP 222, and microprocessor 202 can serve as data 
processing devices. The wireless communication device could be inplemented with a sub 
set of those three units. 
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^^^^y r>;op^». Tmaae E r^'^-r According to Rti Fmbodiment of the 
Invention 

FIG. 3 shows a block diagram of an image encoder 300 according to an 

embodiment of the invention. 

Functional block 302 is a signal deconqxjser. The signal decomposer is preferably 
a multUevel wavelet transformer. A wavelet transformer is known to yield high fideUty of 
a encoded and decoded image to an original image. Another reason which makes the 
wavelet transformer useM in connection with the present invention is that ft is based on 
basis fimctions which are localized in a spatial domain the image, i.e. they are compactly 
supported. The signal decon^ser receives an input image from an image data source 
which can be a directory on a hard drive (not shown), or the image acquisftion device 224 
(FIG. 2). The signal decomposer outputs a set of transformed data coeflScients which can 
be formatted in a 2-d array (matrix). Each color chamiel of an image may be processed 
separately by the signal decomposer and other parts of the encoder 300 and decoder 400 
(HG. 4). Color channels as used in the present appUcation can include but is not limited to 
one of the foUowing luminance (y), chrominance (Cr.Q,). 

Functional block 304 is a spatial activfty analy^r, which serves to obtain a set of 
measures of the variations in a color chamiel over a set of image areas. The set of areas 
can be discrete in the sense that each measure of variation appUes strictly to a region with a 
sharp boundary. The variations may be characterized by one or a plurality of different 
length scales. According to specific embodiments disclosed below the spatial activity 
analyzer deduces measures of variation from a selected set of wavelet transformed data 
coefficients. Each wavelet transformed data coefficients characterizes variations at one or 
more length scales. Operations involving one more wavelet transformed data coeflScients 
can be used to obtain an indications of variations at longer length scales. 

Functional block 306 is a spatial prioritizer/selector, which serves to select a 
phirafity of subsets of a set of transformed data coefficients , and/or deduce an ordering for 
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a plurality of subsets of coefficients. THe selection and/or ordering is based on the 
n^asuresofvariationobtainedbythespatialactivityanalyze. Each subset of transformed 
data coefficients that is selected can correspond, i.e., be spatially correlated, with the set of 
areas over which the spatial activity analyzer obtains measures of variation. 

Functional block 308 is a spatial coefficient grouper which serves to coUect each of 
the plurality of subsets of the set of transformed data coefficients into a data structure 
which we term a 'norm plane' or a 'count plane', in general a priority plane. 

Functional block 310 is a bit stream formatter, which serves to prepare the norm or 
count planes for transmission or recording. In addition to the norm or count planes, th^ 

stream formatter also receives another data structure comprising a coUection of 
transformed data coefficients corresponding to low spatial frequencies. The same 
transformed data coefficients are also distributed among the norm planes. Each low spatial 
frequency transformed data coefficients ^ once in the collection of the same, and 
again a spatially corresponding norm plane. THe bit stream formatter 3 1 0 outputs 
formatted encoded data to the channel only if desired chamiel rate and/or SNR 
requirements are met at decision btock 311. Otherwise, fiirther processing is achieved by 
entropy coder 312. 

The coUection of low spatial frequence coefficients, and norm or count planes are 
serialized and interleaved with demarcation markers in a bit stream. The demarcation 
markersfecffitateparsingthebitstreambyadecoder. The coUectton of transformed data 
coefficients corresponding to low spatial frequencies wiHbe placed into the bit streamer 

so that it will be received by a receiving decoder first. After receiving the collection of 
transformed data coefficients corresponding to low spatial frequencies, the decoder can 
reconstruct a low fidelity verston of the image. 

Functional block 312 is an entropy encoder which can be an arithmetic coder or a 
huffinanencoder. Entropy encoders are known in the art. and will not be described herein. 

In the case of audio signal applications of the invention, the spatial activity 304 
would be replaced by a temporal activity analyzer since the natural domain of an audio 
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signal is time rather than space coordinate. Other fimctional blocks in the encoder 300 
would also have corresponding temporal analogues in the case of audio signal processmg. 

In the case that an image encoder 300 is implemented on the wireless 
communication device 1 1 8 (FIG. 1 ), the transceiver (2 1 6 FIG. 2) can be used to transmit 
the bit stream generated by the bit stream formatter 310, or an output of the entropy 
encoder if that is also implemented in the communication device 1 1 8 (FIG. 1). 

In the encoder 300, note the absence of a quantizer and an entropy coder preceding 
the bit stream formatter 310. Tb^ invention achieves high fideUty in consideration of the 
byte size of the norm planes encoded and decoded, by inter alia grouping the coefficients 
into norm planes and ordering and/or selecting a phuality of the norm planes. Therefore, 
quantizing in order to further reduce the byte size and reduce the image transmission tmie 
can be eliminated. Dispensing with the quantizing has the additional advantage vis a vis 
time considerations that the computationally intensive task of dequantizing, which in the 
case a wireless system is likely to be performed by a low clock rate low processing power 
communication device can be eliminated. For appUcations involving low processing power 
wireless commumcation devices 118, and higher bandwidth, the entropy encoder 312 
could be dispensed with to avoid the necessity of carrying out entropy decoding by the 
wireless communication device 118. 

In the prior art, entropy coders operate on the entire quantized decomposed 
image, thus, making the conq)utational intensity of these entropy coders to be high. The 
advantage of system 300 is that the simple low intensity computational activity analyzer 
allows either: 

1) a system that does not require entropy coding or any fiirther coding; 
such as for wkier bandwidth channels; or 

2) less complex entropy coders for systems with narrow bandwidth 
channels. 

As an alternative to nrnning software embodying the fimctional blocks shown in 
FIG. 3 on a single computer, preferably the image source server 102 (FIG. 1), they may be 
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distributed across multiple computers. An alternative distribution is as Mows. The signal 
decomposer may be run on the image source server 1 02 (FIG. 1). Alternatively the signal 
decomposer can be run another computer (not shown) which receives data from an image 
input device (e.g., digital camera, flat bed scanner, (not shown)), and the output of the 
5 signal decomposer then transferred to the image source server 102. The remaining 

functional blocks shown in FIG. 3 may be run on the wireless network gateway server. In 
this distributed arrangement fiirther fimctional blocks such as entropy encoders and 
decoders, quantizers, dequantizers and network communication processes (not shown) can 
intercede between the signal decon^ser 302 and the spatial activity analyzer. The 
10 distributed arrangement has the foUowing advantage. A wireless Internet service provider 
can implement functional blocks 304, 306, 308, and 310 on a wireless network gateway 
server 108 which they own in order to provide high quality and reduced byte size images 
(hence fester downloads) to their customers, without imposing limitations what Internet 
servers (represented by image source server 102) their customers can access. The 
1 5 foregoing arrangement constitutes a transcoder. 

mnrlc nia pram of an Tm^ pe necoder According t o an Embodiment of the 
Invention 

20 Referring to FIG. 4 a block diagram of an image decoder 400 is shown. 

An entropy decoder 402 receives an entropy encoded bit stream output by entropy encoder 
312 (FIG. 3) and performs the corresponding inverse process to obtain a dupUcate of the 
bit stream generated by the bit stream formatter 310 (FIG. 3). 

A bit stream parser 404 separates out the collection of transformed data 
25 coefficients corresponding to low spatial frequencies, and the norm or count planes. In 
doing so the bit stream parser 404 may employ the aforementioned demarcation markers. 

A spatial coefficient reorganizer 406 arranges coefficients extracted from the norm 
plane into an arrangement suitable for input to the inverses signal deconq)Oser 408. 
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Aocording to one emlxxiimmt of the invents described below in more detail, a 
»m,plane order isdeteminedby the spatialprioritfeer 306 (FIG. 3)inaccordat«ewnh 

measuresofvariation over differer* areas «l»h is based onanalysisofthecoB^^ 
transfonned data coe&rentscorresporrfing to tow spatial 4e<,«»ci«^ tathefcregomg 
erix>diment, the spatial reorganizer 406 deduces which image area each norm plane .s 
associated with, by performing the san« analysis as the spatial activity analyzer, 304 (FIG. 
3) and spatial prioritizer 306 (FIG. 3). Hence, the norm planes could be formatted mto the 
K,^ in an order vrf&h depends on their associated n^asuie of activity, and te^ 

their importance in eom^ image informaaon (as to opposed to a simple raster hke 
order), and yet the decoder would be able to determine the proper spatial assocanon of 

each norm plane. 

Process btock 408 performs as an inverse signal decomposer using coefBcients 
extracted ftomthenormpbnes as input. Inverse signal decomposers are known to 
personsofordinaryskillintheimageprocessingart. According to the instant invention, « 
is preferred that the inverse signal decomposer process the coUection of transformed data 
coefficients corresponding to low spatial frequencies first, in order to obtain a k-w 
resotationimage for display in a short time. Subsequently, fl» inverse signal decon^oser 
can process successive norm planes in the btt sBeam to obtain progr^ively higher unage 
resolution- 

Parts of the encoder 300 (FIG. 3), and decoder 400 (FIG. 4)can be in^lemented m 
the form of software which can be stored on computer readable medium 120 (HG. 1), 
ROM 206 (FIG. 2), or temporarily in RAM 204 (HG. 2). or they can be implemented by 
IP ASIC 220 (FIG. 2) or IP DSP 222 (FIG. 2). 

p„ti^ Qfr„.tur^ of Coeffiri^t^ "iit pnt bv Sipnal Decomposer 

FIG. 5 is a schematic representation of a 8N by 8M matrix of transformed data 
coefficients 500 output by a three level discreet wavelet type signal decomposer which 
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according to the preferred embodiment is the type of signal decomposer 302 used. The 
form of the schematic sho>vn in HG. 5 is known to workers of ordinary skill in the image 
processing art. The schematic shows 10 sub band sub matrices labeled with underlined 
reference numeralsO-9andhavingrectangularbordersasseenintheFIG. Subbandsub 

matrix 0 contains coefficients corresponding low spatial frequency information. The 
coordinates (index pairs) of the upper left points of some of the sub matrixes are written 
along the top and left edges in terms of N and M. From these, the coordinates of the 
upper left points of the remainmg sub matrices can be construed. 

According to an embodiment of the invention sub matrix 0 is divided into blocks of 
two by two transformed data coeflScients. Each of block of two by two coefficients is 
spatially correlated to a particular block of pixels of an input image from which the 
coefficient matrix is derived by applying the wavelet type signal decomposer. For example 
in the case of 3 level wavelet transform each block of 2 by 2 transformed data coefficients 
in sub band sub matrices 0,1,2, or 3 corresponds to an a block of 16 by 16 pixels in the 
input image. Furthermore, according to the preferred embodiment of the invention each 
block of 2 by 2 transformed data coefficients from the 0 sub band sub matrix is grouped by 
the spatial coefficient grouper 308 with spatially correlated transformed data coefficients 
selected from all of the other sub band sub matrices. The latter grouping of coefficients is 
the aforementioned norm plane according to the preferred embodiment. A specification of 
the members of anormplane selected from the coefficient matrix 500 foUows. A block of 
2 by 2 coefficients from the 0 sub band sub matrix can be identified by the matrix indices of 
its upper left member written as (n,m) where n indicates the column of the transformed 
data coefficient, and m indicates its row. In the following specification the transformed 
data coefficients which con^rise a norm plane associated with transformed coefficient 
(n,m) in sub band sub matrix 0 are indicated in terms of n and m referenced with respect to 
thl upper lefthand member, of the sub band sub matrix in which they appear. (As opposed 
to being referenced with respect to the upper left hand transform coefficient of the whole 
matrix 500.) 
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10 



For 0,1,2, and 3 sub band sub matrices, the transformed data coefficients are 

[(n,m), (iH-1^), (lyn+l), (n+l,m+l)], 
for the 4, 5, and 6 sub band sub matrices, the transformed data coefficients are 

[2n-l:2n+2, 2m-l:2m+2], and 
for the 7,8, and 9 sub band sub matrices the transformed data coefficients are 
[4n-3:4n+4, 4m-3:4m+4]. 

For each sub band sub matrix the expression for the transformed data coefficients 
that are in the normplane are .vritten in termsofnandmwhere the upper left member of 

the sub band sub matrix is taken as (1,1) for the purpose of the expressions). The mdices 
ofthese upper left hand membersofeach sub band sub matrix, in the index system for the 

entire transformed data coefficient matrix 500 is given in terms in capital N, M and can be 

construed from FIG. 5. 

Thus for each of the 4. 5, and 6 sub band sub matrices, there is a block of 4 by 4 
transformeddatacoefficients, and for eachofthe7,8and9sub band sub matrices there is 

a block of 8 by 8 transformed data coefficients. Each norm plane can be arranged as a 1 6 
by 1 6 norm plane matrix or can be serialized (in a predetermined order) into a stream of 
256 transformed data coefficients, for the case of a three level discrete wavelet transform. 

Flnw Diagram far S patial Ac tivity Analyzer 

Referring to FIG. 6 a flow diagram of a process 600 performed by the spatial 
activity analyzer 306 (FIG. 3) is shown. In process block 602 the matrix 500 (FIG. 5) of 
transformed data coefficients output by anN level wavelet type signal decomposer (an 
embodiment of signal decomposer 302 (HG. 3)) is read. 
25 In process block 604 for each block of two by two transformed data coefficients m 

sub band sub matrix 0 a set of three differences are taken between the upper left hand 
transformed data coefficient and the other three transformed data coefficients in the block 



15 



20 
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of by two. The cboice to include te upper left transformed data coeffieien. u, all 
three diffirences as opposed to one of the other three is arbitrary. 

In process block 606 for each bk-ck oftwo by two transformed data coefBoents, 
fl«n^«nofthethreedi0ereneescon>putedinthep«vk,usstepisde.er.n«»d. 

In process bk-ck 608 records comprising the maximum for each 2 by 2 Hoek m 
associatton with indexes identi^tag the two by two btock are stored in a list. 

In the case of cotor images the matrix 500 which the spatial activity analy2«r 
a„,,^w„ddcorrespo.«itoonep«rticutaeotorchan.«L Ttee will be separate 
natrices 500 for each cotor channel and each wiD be processed separately. 

r„^-^~.i iwri piion of Smtial P rioritize /Selector 

Referring to TIG. 7 a flow diagram of a process 700 performed by the spatial 
prioritizer m 306 (FIO. 3) is shown. In process block 702 a subset of r«ords from 
tt„ list of indfces and associated maxima that was stored in process block 608 (FIG. 6) ts 
selectedonthebasisofthevaluesofthemaxim. All fl» records for which the maximum 
exceedsapredeterminedthreshoklcanl* selected. AHematWely. the firs, preselected 

number of records taken in descending order according the maximum vah« eanbe 
selected. It is also possible that process bbck 702 is eiftctively eWnat^l ^ 

subset to he equal in size to the number of records in the list. 

In process bk«k 704, the subset of records selected in process block 702 .s sorted 
accordingtothevataesofthemaxima.Aquicksortalgori.hmcanbeused.operform.he 

sort. 

The selecfon performed in process btock 702, and the sort performed process 
i btock 704 are later used .o determine which norm planes will be sent to a decoder and m 
what order. Ordering the normplanes by itself i..u:es.ha.U>c information which ismore 

taportant in determining in«ge quality is received by *e decoder first so .ha. .he decoder 
canpresentabetterqualityimage fester. Setecting norm planes by itselt in««s that the 
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decoded image will have high quaUty relative to the byte size of the image encoded, 
transmitted, and decoded, so that high quality images can be transmitted quickly over a 
low bandwidth data link, and decoded quickly by a low processing power communication 
device. 

5 

Fnnrtinnal Description of Spatia' rnftfficient Grouper 

Referring to FIG. 8 a flow diagram of a process 800 performed by the spatial 
coefficient grouper 308 (FIG. 3) according to an embodiment of the invention is shown. 
10 In process block 802 a plurality of subsets of transformed data coefficients are 

collected together from the transformed coefficient matrix 500 (FIG. 5). The plurality of 
subsets corresponds by inclusion of the identified two by two block to the subset of 
records selected by the spatial prioritizer selector 306 (HG. 3) in process block 702 (FIG. 
7). Each subset (norm plane ) comprises a block of two by two transformed data 
15 coefficients from sub band sub matrix 0 and a coUection of spatiaUy correlated transformed 
data coefficients from the remaining sub band sub matrices 1-9 of transformed data 
coefficient matrix 500. The set of spatially correlated transformed data coefficients for a 
given two by two block in sub band sub matrix 0 of a 3 level wavelet transform is specified 
above in the description of FIG. 5. 
20 In process block 804 the norm planes are stored in memory in order following an 

order derived for the two by two blocks by the spatial prioritizer/selector 306 (FIG. 3) in 
process block 704. 

The processes performed by the spatial activity analyzer 204 (FIG. 2) and the 
spatial prioritizer/selector 206 (FIG. 2) can be reorganized and the results of these process 
25 achieved in an alternative way. 
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Fimrfinnfll Description Formatter 

Referring to FIG. 9 a flow diagram of a process 900 performed by the bit stream 
formatter 310 (FIG. 3) according to an embodiment of the invention is showa 

In process block 902 a request for an image at a specified quality level is received. 
The request could for exan^le take the form of a Hyper Text Transfer Protocol (HTTP) 
request. The Universal Resource Identifier (URI) specified in the HTTP request could 
point to a common gateway interfece (CGI) program which serves as a communicatk,n 
module for the bits stream formatter. The URI could include appended data which 
specifies the quality level by key word or number. THe URI could be carried byaWireless 

AppUcation Protocol (WAP) request between the wireless communication device 1 1 8 
(FIG. 1) and the wireless gateway server 108 (FIG. 1). 

In the case of a transcoder implementation the bit stream formatter 310 could 
automatically select the number of norm planes based on information on available 
bandwidth and the display 208 resolution. 

In process block 904 the quality level and/or system bandwidth is translated into a 

number of norm planes to be sent to a requesting communication device 118 (FIG. 1, 2). 

For example the quaUty level could be specified as a number between 1 and 5 and fix>m this 

number the fraction of a total number of norm planes stored in process block 804 (HG. ^ 

to be sent cab be determined. 
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For example, by the following formula: 
NPS = TNN*(QL/5) 
where NPS is the number of norm planes sent 

TNN is the total number of norm planes stored in process block 804 (FIG. 8) 

QL is the quaUty level received in process block 902. 

In process block 906 the 0 sub band sub matrix (FIG. 5) is read. 

In process block 908 the number of norm planes as determined in process block 
904 are read from the memory in which they were stored in process block 804 (FIG. 8) 

In process block 910 the transformed data coefficients in sub band sub matrix 0 and 
the number of norm planes are serialized. 

In process block 912 the transformed data coefficients from sub band sub matrix 0, 
and the subsets of transformed data coefficients from the norm planes are interleaved with 
demarcation markers in a bit stream. The transformed data coefficients from the 0 sub 
band sub matrix are placed into the bit stream first, so that they will be received first by the 
decoder. 

The bit stream produced by the bit stream formatter 310 (HG. 3) can then be 
passed to lower level communication services, for transmission over a communication 
channel, e.g., through the wireless network 11 2 to a wireless communication device 1 1 8. 

Note that the process accomplished by the spatial activity analyzer 204, spatial 
prioritizer 206 and spatial coefficient grouper 208 (all in FIG. 2) is usefid without encoding 
the norm planes into a bit stream. THe norm planes need not be transmitted in seriaUy they 
can for example be sent using a paraUel data link for example. The norm planes need not be 
transmitted at all, they can be stored as matrices or otherwise on a CDROM for example. 
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T7W niflf rram of '' Stream Parser 

Referring to FIG. 10 a flow diagram of process 1000 performed by a bit stream 

parser 404 (FIG. 4) is shown. 

In process block 1002 sub band sub matrix 0 (FIG. 5) is read fix,m the bit stream. 

In process block 1 004, sub band sub matrix 0 is forwarded to the spatial coefficient 
grouper 406 (FIG. 4). In process block 1006 successive subsets of transformed data 
coefficientscorrespondingtonormplanesarereadfromthebitstream In process block 

1008 successive sets of transformed data coefficients are forwarded to the spatial 
coefficient reorganizer 406 (HG. 4). Note that a group of one or more successive subsets 
of transformed data coefficients can be read out ofthe bit stream before the group is 
forwarded to the spatial coefficient reorganizer. 

Because the norm planes with the highest activity are entered into the bit stream 
and extracted first, and because they are in general spatially distributed around the image, 
in case a comiection through which the encoded image is broken, the fideUty of the 
decoded image based on the norm planes received up to the break will be higher and will 
not contain a distinct horizontal boundary between a high fidelity area and a lower fidelity 



area. 



VUr^ nia pram of a Soat inl r^ff"^'--^ Ppnrpani/Br 

Referring to HG. 11 a flow diagram of a process 1100 performed by a spatial 

coefficient reorganizer 406 (FIG. 4) is shown. 

In process block 1 1 02 sub band sub matrix 0 (FIG. 5) is received from the bit 
stream parser 404 (FIG. 4) after having been forwarded in process block 1004 (HG. 10). 

In process block 1 104 process 600 (FIG. 6) performed by the spatial activity 
analyzer 304 (HG. 3) and process 700 (FIG. 7) performed by the spatial prioritizer 
selector 306 (HG. 3) is repeated by the spatial coefficient reorganizer 406. The purpose 
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of doing so is to determine on the decoder side the order in which the subsets of 

transfonned data coefficients (norm) planes were encoded into the bit stream. By doing so 

the spatial association of each norm plane is deduced because sub band sub matrix 0 has a 

definite spatial arrangement which is preserved in transmission. 
5 In process block 1 106 successive norm planes which were forwarded by the bit 

stream parser 404 (FIG. 4) in process block 1008 (FIG. 10) are received by the spatial 

coefficient reorganizer 406. 

In process block 1 108 successive norm planes are forwarded to the inverse 

projective transform along with information indicative of each norm planes spatial 
10 associatioa 

With the information indicative of the spatial association, coefficients extracted 
from each norm plane can be correctly placed in an input matrix for the inverse projective 
transform. 

15 Fnnotinnal Descri ptinn nf « Snatial Activitv Analyzer According to a Second 

Embodiment 

Referring to FIG. 12A a process 1200 performed by a spatial activity analyzer 304 
(FIG. 3) according to a second embodiment of the invention is shown. In process block 
20 1202 for each 8 by 8 block of transformed data coefficients in the 7, 8, and 9 sub band sub 
of three color channel specific, three level wavelet transform coefficient matrices 500 (FIG. 
5), a count of the number of transfonned data coefficients for vMch the absolute value 
exceeds a first predetermined threshold is obtained 

In process block 1204, for each of the aforementioned 8 by 8 blocks, the count of 
the number of transformed data coefficients that exceeded a predetermined count threshold 
is determined. 

The first predetermined threshoU value and the predetermined count value can be 
set independently for each color chamiel in the case of color images. In inq>lementations 
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carried out to date by the mventors, they have been set to the same two values fo^ 
color channels. 

The first predetermined threshold value and the predetermined count threshold can 
be adjusted to afect the fidelity of the encoded and decoded image to the original and the 
byte size of the encoded image. Decreasing the either threshold will increase the fidetity, 
but also increase the byte size of the encoded image. 

The table below gives some examples of subjective image quaUty obtained with 
various combinationsofchoicesoffirst predetermined threshold value a^^ 

count threshold. 



FIRST 

PREDETERMINED 
THRESHOLD 



50_ 
25 
16 
16 



PREDETERMINED 

COUNT 
THRESHOLD 



60 
5 
5 
2 



OUTPUT 
BYTE SIZE 
WITH 

REFERENCE 
INPUT 

='18 VGA 
COLOR IMAGE 

4300 
8004 
8845 
10670 



SUBJECTIVE 
QUALITY 



ACCEPTABLE 
V ERY GOOD 
VERY GOOD 
EXCELLENT 



In process block 1206 each 8 by 8 block for which the count exceeded the 
predetermined count threshold is flagged, Le., a record is made of it. 

A representation of the record can take the form of a binary sequence in which 
1 5 each digit corresponds to a particular 8 by 8 block, and the value of the digit indicates 

whether the 8 by 8 block was flagged. This sequence is termed a flag sequence. Accordmg 
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to an alternative preferred embodiment there is a separate flag sequence for each color 
channel. 

In process block 1208 a plurality of records each comprising identifiers for each 8 
by 8 block and a count for each 8 by 8 btock are stored in memory. THe records can be 
stored sequentially after each count is made. 

Turning now to FIG. 13, the process 1200 continues with process block 1210 in 
which the plurality of records stored in process block 1208 are read. 

In process block 1212 the plurality of records are sorted by count to obtain a 
sorted plurality of records. A quick sort algorithm could be used. 

In process block 1214 a sorted list of 8 by 8 block identifiers from the sorted 
plurality of records is extracted. The identifier for each 8 by 8 block inthe sorted ph«-ality 
of 8 by 8 blocks can take the form of a pair of indexes corresponding to the upper left 
member of the 8 by 8 block in index system of the transformed data coefficient matrix 500 

(no. 5). 

Note that rather than analyzing the 8 by 8 blocks in sub band sub matrices 7, 8, and 
9, the same type of analysis could be based on other blocks in other sub band sub matrices. 
For example 4 by 4 blocks of sub band sub matrices 4,5, and 6 can be examined. 

The values first predetermined threshold and/or the predetermined count threshold 
can varied across the image. For example lower values can be used towards the center of 
the image which is likely to be of greater interest. Alternatively as discussed in connection 
with FIG. 1 8 below the values can be varied in accordance with a user specified region of 
interest. 
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rnnrtinnni r^-^ r-- ^^..iM Prioriti7er/Sdector Accord in p to an alternative , 
pi-p Wed Kmbodiment 

Referring to FIG. 14, a flow diagram of 1400 of a process performed by the spatial 
prioritizer/selector 306 (FIG. 3) according to the alternative preferred embodiment of the 
invention is shown. 

In process block 1402, the 8 by 8 block in the 7,8, and 9 sub band sub matrices of a 
luminance component transformed data coefficient matrix 500 (FIG. 5), which were 
flagged in process block 1206 are selected. 

In process block 1404 all the transformed data coefficients in the 1 -6 sub band sub 
matrices of the luminance component transformed data coefficient matrix 500 (FIG. 5) are 
selected. 

In process block 1406 for each of two chrominance transformed data coefficient 
matrices 500 (FIG. 5), for each flagged 8 by 8 block a spatially correlated 4 by 4 blocks 
from the 4,5, or 6 sub band sub matrices is selected. 

In process block 1408 for each of the two chrominance transformed data 
coefficient matrices 500 (FIG. 5), for each flagged 8 by 8 block , a spatially correlated2 by 
2 blocks from the 1,2, or 3 sub band sub matrices is selected. 

•nius in the case of the two chrominance transformed data coefficient matrices 500 
(FIG. 5), if an 8 by 8 block in sub band sub matrix 7 (HG. 5) is flagged then it will be 
selected 'afong with a spatial corresponding 4 by 4 block in sub band sub 
2 by 2 block in sub band sub matrix 1 . Similarly blocks in sub band sub matrices 3 and 5 
will be selected depending on the selection of spatially correlated blocks in sub band sub 
matrix 8; and the selection ofblocks in sub band sub matrices 1 and 4 will depend on the 

selection of spatially correlated blocks in sub band sub matrix 7. 
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^p,tiiii ^-^ffl^i^nt flrmiper Acr^-^i^ p to the altem^tiA^e preferred Embodiment 

Referring to FIG. 15, a process 1500 performed by the spatial coefificient grouper 
308 (FIG. 3) according to the alternative preferred embodiment of the invention is shown. 

In process block 1502 for the luminance component transformed coefficient matrix 
500, each 2 by 2 block from sub band sub matrix 1 is grouped with a spatially correlated 4 
by 4 block from sub band sub matrix 4 and with a spatiaUy correlated 8 by 8 block from 
sub band sub matrix 7 if the 8 by 8 btock was selected by the spatial prioritizer 306 in 

process 1300 (HG. 3) 

In process block 1 504, process block 1 502 is repeated with the following sub band 

sub matrix replacements: 3 for 1, 6 for 4, and 9 for 7. 

In process block 1506, process block 1504 is repeated with the following sub band 
sub matrix replacements: 2 for 3, 5 for 6, and 8 for 9. 

In process block 1 508 for each chrominance component transformed data 
coefficient matrix 500, each 8 by 8 block in sub band sub matrix 7 is grouped with a 
spatially correlated 4 by 4 block from sub band sub matrix 4 and a spatially correlated 2 by 

2 block in sub band sub matrix 1 . 

Inprocess block 1510, process block 1508 is repeated with the folfowing sub band 

sub matrix replacements: 9 for 7, 6 for 4, and 3 for 1 . 

In process btock 1512. process block 1510 is repeated with the foUowing sub band 
sub matrix replacements: 8 for 9, 5 for 6, and 2 for 3. 

The groups of coefficients formed in process blocks 1502-1512 are spatially 
correlated subsets of transformed data coefficients, termed 'count planes'. 

Note that, the luminance component is treated differently than the two chrominance 
components by the spatial prioritizer selector 306, and the spatial coefficient grouper 308 
according to the processes shown in HGS. 14 and 15. According to the alternative 
preferred embodiment, sub band sub matrixes 1 to 6 will always be sent for the luminance 
be selected, but in the case of the two chrominance conqwnents they wiU only be sent if 
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the spatially correlated 8 by 8 blocks in sub band sub matrices 7-9 meet the criteria of the 
spatial activity analyzer 304 (FIG. 3) as determined in process 1200 (FIG. 12). This 
distinction is based on an understanding of human visual perception, from which it is 
known that the human eye is more sensitive to changes in luminance as opposed to 

5 chrominance. 

Note that as each group is formed it may be stored in memory before another 
group is formed. 

pu c^o». v.m^^e^ Accorf^^ r ^" ^^^"^^^^^ "^^^^""^^ Fmbodiment of the 
10 Invention 

Referring to FIG. 16, a process 1600 performed by a bit stream formatter 310 
(HG. 3 according to the alternative preferred embodiment of the invention is shown. 

In process block 1602 the transformed data coefficients from sub band sub matrices 
15 0 from the luminance and the two chrominance transformed data coefficients matrices 500 
are read. 

In process block 1604 the transformed data coefficients read in process block 1604 
are serialized. 

In process block 1606 the bit sequences are written into a bit stream in a 

20 predetermined order. 

In process block 1608 the sorted list of 8 by 8 identifier which had been extracted 
by the spatial activity analyzer 304 (HG. 3) in process block 1214 (FIG. 12B) is read. 
In process block 1610 the sorted Ust of 8 by 8 block identifiers is serialized. 
In process block 1612 the serialized list of 8 by 8 block identifiers is written into 

25 the bit streantL 

In process block 1614, successive count planes are read in order according to the 

sort order obtained in process block 1212. 

In process block 1616 successive count planes are serialized. 
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In process block 1618 successive serialized subsets of spatially correlated 
transformed coefficients are written into the bit stream. 

Thus, according to the alternative preferred embodiment, the count planes will be 
written into the bit stream in order according to the count of the number transformed data 
5 coefficients in each 8 by 8 block that exceed the first predetermined threshold. 

Different color chamiel subsets can be blocked together in the bit stream or 
interleaved. 

F,T^.^»..i n.«^ ption of ^ ^*r..m Parser Arrordinp to the alternative 
10 preferred Embodiment 

Referring to FIG. 17 a flow diagram of a process 1700 performed by the bit stream 
parser 404 (FIG. 4) according to an embodiment of the invention is shown. 

In process block 1702, transformed data coefficients from sub band sub matrix 0 

15 are read from the bit stream. 

In process block 1704, the transformed data coefficients from sub band sub matrix 

0 are forwarded to the spatial coefficient reorganizer 406 (HG. 4) 

In process block 1 706 the sorted list of 8 by 8 block identifiers is read from the bit 



20 



stream. 

In process block 1708 the sorted list of 8 by 8 block identifiers is forwarded to the 
spatial coefficient reorganizer 406 (FIG. 4) 

In process block 1710 successive count planes are read from the bit stream. 

In process block 1712 successive count planes are forwarded to the spatial 
coefficient reorganizer. 



25 
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Fnnrtinnn' n.c^ ptinn of « Process Perfo rm ed by the Spat MCoefficjent 
pp^rpo»;^.r Ar^^rAina to t b'- ^it^m^tive nreferred Embodiment 

Referring to FIG. 18, a flow diagram of a process 1800 performed by the spatial 
coefficient reorganizer 406 (HG. 4) according to the alternative preferred embodiment is 
shown. 

In process block 1802 transformed data coefficients from sub band sub matrix 0 are 
received from the bit stream parser 404 (FIG. 4). 

In process block 1804 the transformed data coefficients from sub band sub matrix 0 
are formatted for the inverse projective transform 408, FIG. 4. 

In process block 1 806 the transformed data coefficients from sub band sub matrix 0 
are forwarded to the inverse projective transform 408, FIG. 4. 

Inprocess block 1808 the sort list of 8 by 8 block identifiers is received from the 

bit stream parser 404 (FIG. 4). 

In process block 1810 successive count planes are received from the bit stream 

parser 404 (FIG. 4) 

In process block 1812, successive count planes are forwarded to the projective 
inverse transform 408 (FIG. 4) along with information identifying each count plane that is 
deduced from the sorted list of 8 by 8 block identifiers. As each successive count plane is 
received, the spatial coefficient reorganizer can deduce by examinmg the identification of 
each successive member of list of 8 by 8 block identifiers. Knowledge of the spatial 
correlation between each 8 by 8 block from sub band sub matrices 7, 8, 9 and other blocks 
from other sub band sub matrices is programmed into the spatial coefficient reorganizer 
406 (FIG. 4), so that it can identify other transformed data coefficients in the count planes 
in addition to the transformed data coefficients from the 8 by 8 blocks and properly 
identify those additional coefficient to the inverse projective transfomi 408 (FIG. 4). Note 
that the information identifying the count planes to the inverse projective transform may 
not be expUcit. It can be impUed in the positioning coefficients extracted from the count 
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plane in a matrix in Avhich they are placed by the spatial coefficient reorganizer for 
processing by the inverse projective transform 408 (FIG. 4). 

P„„.^i^..i n^c^ption of Re r»" Tntfirest Based Prioritization 

HG. 19 shows an optional process 1900 for the spatial activity analyzer 304 for 
prioritizing use of bits in encoding an image based on a user mput region of interest is 
sho^vn. The process is particularly suited for applications in which the encoder 300 (FIG. 
3) is run on a computer for converting bit map images, in which case the encoded image 
may be stored, e.g. on the image source server 102 (FIG. 1 ) rather than being transferred 
immediately. 

For the purpose of implementing the process shown in FIG. 19, the image encoder 
would be provided with a graphical user interfece (GUI) which a user could use to view an 
image and select one or more areas regions of interest. The GUI could for example allow 
the user to size and position a rectangle over an area of the image in order to select it as a 
region of interest. Providing this type of GUI is within the ability of a person having 
ordinary skill in the GUI programming art. 

Referring to FIG. 19, in process block 1902, a user specification of a region of 

interest is read. 

In process block 1904 a value of the first predetermined threshold to be used within 
the ROI is set. 

In process block 1906 a value of the count threshold to be used within the ROI is 



set 



The vakes set in process block 1904, and 1906 are different from other values 
which can be default values which are used outside the ROI. 

Generally speaking either the first preselected threshold, the count threshold, or 
both will be lowered in the ROI. The two thresholds may or may not be increased outside 
of the ROI. 
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Lowering the first predetermined threshold in the ROI will increase the count for 8 
by 8 blocks in the ROI and improve the positioning of count planes corresponding to the 
ROI in the sorted list extracted in process block 1214 (FIG. 13) 

Lowering the count threshoM to be used in the ROI will increase the number of 8 
by 8 blocks in the ROI that are selected in process 1 500 (FIG. 15) 

Pi,».f;»nal nPfu-Ti ption of Vo'-i^M^ SNR Embodiment 

FIG. 20 shows a process 2000 by which an image represented by transformed data 
coefficients which is arranged in a set of norm or count planes can be accessed at a 
selected signal to noise ratio (SNR) level. THe process can be implemented on the encoder 
300 (FIG 3)oritmaybeimplementedonthedecoder400(FIG.4)skie. In the former 
case it can be implemented in the bit stream formatter 310 (FIG. 3),,which would selected 
certain transformed data coefficients to encode. In the latter case it can be implemented by 

the bit stream parser 404 (FIG. 4). 

Referring to FIG. 20, in process block 2002 a request for an image at a specified 

quality level is received, e.g., as a subroutine call. 

In process block 2004 based on the specified quaUty level an upper sub band limit 
is determined. For example for thumb nail quaUty the upper sub band limit wiU be set at 
zero. In the case of a three level wavelet transform medium quality could correspond to an 
upper sub band limit of 3 or 6. For the highest quality, the upper sub band limit would 
correspond to the highest sub band-sub band nine in the case of a three level wavelet 
transform. 

In process block 2006 a set of sub band portk>ns are read firom an ordered set of 
count ornormplanesofthe image up to the upper sub band limit to form an ordered setof 

sub band limited count or norm planes. Process block 2006 preserves the ordering which 
is preferably based on the level of activity. 
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In process block 2008 the ordered set of sub band limited count or norm planes is 
suppBed to the inverse projective transform 408 (FIG. 4) 

The term natural domain as used in the present specification means spatial domam 
ortemporal domain. For an audio signal the term natural domain refers to a time domain. 

no. 21 is a flow diagram of a process that can be performed by the bit stream 
formatter 310 (FIG. 3) in order to determine a number of count or norm planes to be 
encoded. Alternatively, the process shown in FIG. 2 1 could be performed by the spatud 
prioritizer selector 306 (FIG. 3) to limit the number of norm planes selected. 

Referring to FIG. 21, in the process block 2102 a compression rate specified m bits 
per pixel or an available system bandwidth or user defined image resolution or image 
quality is read. The compression rate can be a predetermined value that is used m a 
particularwirelessnetwork. THe compression rate can be selected in view of the currently 
available bandwidth to obtain image transport times that do not cause noticeable delay to a 
user Also, the compression rate can be calculated from a specified image resolution from 
which the numberofnormplanescanbe selected by properly arranging the subbandsmthe 

bitstreamofsuchmultiresolutionalsystem. Alternatively, image quality can be specified 
and an approximate compression rate can be selected from a programmed look-up table. 
Based on the codec performance, several test images may be tested a priori and a 
relationship between image quality and con^ression rate be obtained to form the table. 

In process block 2104, a total number of bits to be aUocated for a compressed 
version of an image is calculated based on either a preselected con,,ression ratio, an image 

resolution, or image quality. 

In process block 2106, a total number of norm or count planes corresponding to 
the totalnumberofbitsnumberofcalculated in process block2104 is determined. The 

total number of transformed data coefficients for a norm plane is, according to the 
embodiment described above in connection with FIGs. 6-1 1 , fixed at 256 ultimately, 
depending on the nmnber of wavelet transformation levels. If; as is preferred, the 
transformed data coefficients are not quantized, then the total number of bits in a norm 
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plan.bsmvlyfl«.o.a.n«nteofm>nsfon,»d<.a,acoefflcien.si„a„ormpla«m»»^ 
byU^^^berofbiCsu^dtorepreserteaoh^ansibrn^ddataeoefficien. (8 bits is suimbte 
fornotvscientifc applications). Given the .»»ber of bits per .»nn plane, process block 
2,06 can be accon^lished by dividingthetotaln^berofbitsibr the image, ascd^ 

in process btock 2104, by the number of bits per norm plane. If transformed data 
coefficients are quantized, the number of bKs per norm plane »ill be reduced. 

The invention provides a method and system for selecting and ordering spatially 
correlated subsets of coefficients based on their relative importance in conveying data 
(eg image)infbrn»tionByencodi«a„ddecodingtheselected.and/ors„rtedspa,.* 

correlated subsets of coefficients a higher fidelity (as mdicated for example by SNR) 
representation of the data can be obtained, for a given byte size of the »b sets decoded. 

According to one aspect the inventton provides a method of encoding a data set 
comprising stepsofiperfonningatransfonnonadata set to obtainasetoftransfonned 

data coefficients; forming a ph^ality of subsets oftransformed data coeffictents 
corresponding to a set of tegions of the data set by grouping transformed data coefficients 
ftom the set oftransformed data coefficients; and calculating a measure of actn«ty 
corresponding to each of the phmJity of subsets of transform coefficients. 

According to another aspect the invention provides a computer readable mednm, 

containing programming instractfens for encoding a data set comprising ptogrammmg 
b^ructions for: re^lmgasetoftransforn^l data coefficients corresponding toadata set; 

obtaining a measure of variatk,n over a region of the data set; determining whether to 
extract a subset of tramformed data coefficients that is spatially correlated to the region of 
data set on the basis of the measure of variation; and extracting a subset of transformed 

data coeflScients. 

According to yet another aspect the invention provides a method of processing a 
set oftransformed data coefficients output by a wavelet signal decomposer comprismg 
steps of receiving a set oftransformed data coefficients output by a wavelet signal 
decomposer; computing a set of measures of variation corresponding «, a set of selected 
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^oUt^n^ taagertuuisrep^semedby the se. of transfenneddatacoem 
selectittg a s.*«a of the set of a«as of tl« t,a.rf>rn«d image on the basis of the set of 

of variation; detemining an ordering for the subset of the set of areas on the 
basis of the set of measures of variation; and groupmg a set of subsets of the set of 
5 transformed data coefBcients corresponding to these, of areas of the nnage. 

According to a ftrther aspect the invention provides An encoder for encodmg a 
data set comprising: a signal decomposer for receiving an input data set and outputtmg a 
se,oftransforn«ddatacoei&ien.s;aspatial activity analy^ for determini^ 
nsasures of variation of the data s« on the tasis of the set of transformed data 
,0 coefficien,s;asp^prioriti^mfori«rforminga«n»tionselectedih>mag™^^ 

consisting of selecting a set of subsets of the set of transfonned data coefficients and 
aeterminh^anorferingforaset of subsets ofthesetoftramformeddatacoeificien^ 

a spatial coefficient grouper for collecting together a pluraHty of spatially correlated 

subsets of the set of transformed data coefficients. 
,5 Accordingtoastinfurtheraspect,themve«ionpr„videsad=coderfordecodmg 

an encoded data set comprismg:abit stream par^r for extractingaset of spattally 
correlated subsets of a se, of transformed data coefficients ftom a bit stream; a spa«al 
coefficient reorganizer for forwarding the set of spatially correlated subsets of the set of 
t^sformed data coefficients to an inverse signal decomposer along with an indicatK.n of 

20 the spatial association of each sub set. 

According to a stiU further aspect, the invention provides a communication system 
con^rising: an taage source server comprising:asignaldecomposer for receivingan^^^^ 

data and outpuUing a set of transformed data coefficients; a spatial activity analyzer for 
determining a set of measures of variation of the data set on the basis of the set of 
ttansformed data coefficients; a spatial prioritizer/selector for performing a &nctK,n 
selected from a group consisting of selecting a set of subsets of the se, of transfonned data 
coefficients and determining an ordering for a set of subsets of the set of transformed da^ 
coefficients; and a spatial coefficient grouper for coltecting together a plurality o^ 
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correlated sa»«ts of the s« of trarsfon^d data coefficients; and abft stream fbnmtter for 

™ti„g fl« plurality of spatially correlated subsets into a bit stream; a network; a 
comm»nicatk,n device con^: a bit stream parser for extracting the pluratey of 

spatially correbted subsets item a W. strean. a spatial coefSdent reorgamzer fo^ 
5 for«rdingthepluraayofspa.ianycorrela.eds»bsets.oa.inversesignaldeoomp^^ 

atong with an indication of the spatial association of each sub set. 

According to a stffl tother aspect, the invention provides a method of pnorifzmg 
tta,,^ dam coefficiems output byadi^^tewelettransfonncomp^ 

selecting a block of hmtonce transformed data coefficients fiom a first sub band sub 
,0 natrix in a luminance transformed data coefficient matrix; selectively selectmg a 

corr^ponding block of chrominance transformed data coefficients ftom a correspondmg 
sub ba..! sub n^Urixinachrominancetransforn^d data coefficiem matrix on the basrs Ota 

measure of activity corresponding to the corresponding btock. 

Although the invention has been explained above with reference to flow diagrams 
1 5 the invention is not limited to implemenlation as one or more programs run on general 
pu^se microprocessor base systems. TT« invention can be implemented in whole or m 
p.„, using digital signal processing (DSP) chips, and appUcation spedfic integrated circmts 
(ASIC). 

TlK flow diagrams do not necessarily dictate a specific temporal sequences of 
20 processsteps.Additk>nallytheprocessneednotbebrokendownasshowninthenGS. 
Variations, in addition to tiK-seexpEcifl, mentioned above, are wiUun the SkiU ofpersons 

having ordinary skill in tiie image processing art. 

While the prefer,«l and other embodiments of tiK invention have been illustrated 
and described, it will be clear ti>at the invention is no. so limited. Numerous modifications, 
25 changes,variatk.ns,s„bs.itutions,a»de<puvalentswilloccurtotiK,seofordim^ 

.he art witho« dq^rting ftom ti« spirit and scope of tire pre^n. invention as defined by 

the following claims. 
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